Temporal error concealment for video communications

ABSTRACT

Methods and systems for processing video data are described. A set of candidate motion vectors is selected from motion vectors associated with macroblocks in a first frame of video data and from motion vectors associated with macroblocks in a second frame of the video data. A statistical measure of the set is determined. The statistical measure defines a motion vector for a macroblock of interest in the second frame.

FIELD

Embodiments of the present invention pertain to the processing ofmultimedia data, and in particular to the decoding (decompressing) ofvideo data.

BACKGROUND

Media systems transmit media data, such as video data, over wired and/orwireless channels. Data transmitted over such channels may be lost orcorrupted or may experience delays along the way, perhaps arriving lateat its destination. Late or lost data can be particularly troublesomefor video data that are predictively encoded (compressed) usingtechniques such as but not limited to MPEG (Moving Pictures ExpertsGroup) encoding. Predictive encoding introduces dependencies in theencoded data, so that the decoding of some data depends on the decodingof other data. While predictive encoding generally improves the amountof compression, it can also result in error propagation should datarelied on for the decoding of other data be lost or arrive late. Anylate or lost data can impact the quality of the reconstructed (decodedor decompressed) video data. However, the impact can be aggravated ifthe lost or late data is part of a reference frame used for motioncompensated prediction because errors will propagate to other framesthat are dependent on the reference frame.

For instance, consider a moving object that appears in differentpositions in successive frames of video data. Using predictive encodingtechniques, the object is described by data in the first frame, but inthe second frame the object is described using a motion vector thatdescribes how the object moved from the first frame to the second frame.Thus, only the data for the motion vector needs to be transmitted in thesecond frame, improving the amount of compression because the datadescribing the object does not need to be retransmitted. However, if themotion vector is not received, then the object cannot be properlyrendered when the second frame is reconstructed into a video image, thusreducing the quality of the reconstructed video. Subsequent frames inwhich the object appears may also be affected, because they may dependon proper placement of the object in the second frame.

To alleviate the impact of absent (e.g., missing, lost, late orincorrectly received) data on the quality of the reconstructed video, avideo decoder can apply an error recovery (e.g., error concealment)process to the received data. Studies have shown that the quality of thereconstructed video can be significantly improved if motion vectors canbe recovered (e.g., estimated). Temporal error concealment improves thequality of the reconstructed video by estimating missing or incorrectlyreceived motion vectors in a current frame using properly receivedinformation from the current frame and/or preceding frames. In otherwords, a goal of temporal error concealment is to estimate motionvectors using their spatial as well as temporal associates.

Conventional temporal error concealment techniques are based in thepixel domain. Consider a frame (the current frame) in which a motionvector associated with an area (e.g., a macroblock of interest) in theframe is missing. A set of motion vectors is formed by selecting motionvectors associated with macroblocks that surround the macroblock ofinterest in the current frame and motion vectors associated withmacroblocks that surround the co-located macroblock in the referenceframe (the co-located macroblock is the macroblock that is at the sameposition in the reference frame as the macroblock of interest is in thecurrent frame). With a pixel-domain approach, a measure of distortion iscalculated for each of the motion vectors in the set. To evaluate thedistortion, pixel values are taken from the reconstructed frame buffer.In a motion select technique, the motion vector that minimizes thedistortion measure is chosen as the replacement for the absent motionvector. In a motion search technique, a search for a motion vector thatminimizes the distortion measure is performed within, for example, a 3×3window of macroblocks.

Pixel-domain error concealment is problematic because it iscomputationally complex and time-consuming. Evaluating the distortionfor each potential motion vector can require a large number ofcomputations, consuming computational resources and causing delays inthe decoding process. Pixel-domain error concealment is most effectivewhen performed after the decoder has finished decoding a frame; hence,the delay introduced by error concealment in the pixel domain may beequivalent to one frame duration. Furthermore, accessing thereconstructed frame buffer to retrieve pixel values for the distortionevaluation takes time, which adds to the delays.

Accordingly, a method and/or system that can reduce computationalcomplexity and decoding delays would be desirable.

SUMMARY

Methods and systems for processing video data are described. In oneembodiment, a set of candidate motion vectors is selected from motionvectors associated with macroblocks in a first frame of video data andfrom motion vectors associated with macroblocks in a second frame of thevideo data. In one embodiment, the first frame precedes the second framein order of display. A statistical measure of the set is determined. Forexample, the average or the median of the candidate motion vectors canbe determined. The statistical measure defines a motion vector for amacroblock of interest in the second frame.

Various methods can be used to select the candidate motion vectors. Theselection of the candidate motion vectors and the determination of areplacement motion vector are performed in the motion vector domaininstead of the pixel domain. Consequently, computational complexity andlatency are reduced. As an additional benefit, hardware modificationsare not required.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of one example of a system for decoding videodata.

FIG. 2 illustrates an example of two frames of image data organized asmacroblocks.

FIG. 3 illustrates an example of two image frames showing the motion ofan object from one frame to the next.

FIG. 4 is a data flow diagram showing the flow of data from a dataencoding process to a data decoding process.

FIG. 5 is a flowchart of a motion vector domain-based temporal errorconcealment method.

FIG. 6 illustrates the flow of information according to the method ofFIG. 5.

FIG. 7 is a flowchart of a method for selecting candidate motion vectorsused in a motion vector domain-based temporal error concealment process.

FIG. 8 illustrates the flow of information according to the method ofFIG. 7.

FIG. 9 is a flowchart of a method for detecting frame-to-frame motionchange and used for selecting candidate motion vectors used in a motionvector domain-based temporal error concealment process.

FIG. 10 is a flowchart of another method for detecting frame-to-framemotion change and used for selecting candidate motion vectors used in amotion vector domain-based temporal error concealment process.

FIG. 11 is a flowchart of a method for locating a motion boundary withina frame and used for selecting candidate motion vectors used in a motionvector domain-based temporal error concealment process.

FIG. 12 illustrates the flow of information according to the method ofFIG. 11.

FIG. 13 is a flowchart of a method that uses the trajectory of a movingobject to select a candidate motion vector used in a motion vectordomain-based temporal error concealment process.

FIG. 14 illustrates the method of FIG. 13.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the conceptspresented herein. However, it will be recognized by one skilled in theart that embodiments of the invention may be practiced without thesespecific details or with equivalents thereof. In other instances,well-known methods, procedures and components have not been described indetail as not to unnecessarily obscure aspects of these embodiments.

Some portions of the detailed descriptions, which follow, are presentedin terms of procedures, steps, logic blocks, processing, and othersymbolic representations of operations on data bits that can beperformed in computer memory. These descriptions and representations arethe means used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. A procedure, computer-executed step, logic block, process, etc., ishere, and generally, conceived to be a self-consistent sequence of stepsor instructions leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a computer system or similarelectronic computing device. It has proven convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present embodiments,discussions utilizing terms such as “selecting” or “determining” or“comparing” or “counting” or “deciding” or the like, refer to theactions and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In one embodiment, a computer-usable medium that has computer-readableprogram code embodied therein is implemented. A computer system caninclude, in general, a processor for processing information andinstructions, random access (volatile) memory (RAM) for storinginformation and instructions, read-only (non-volatile) memory (ROM) forstoring static information and instructions, a data storage device suchas a magnetic or optical disk and disk drive for storing information andinstructions, an optional user output device such as a display device(e.g., a monitor) for displaying information to the computer user, anoptional user input device including alphanumeric and function keys(e.g., a keyboard) for communicating information and command selectionsto the processor, and an optional user input device such as a cursorcontrol device (e.g., a mouse) for communicating user input informationand command selections to the processor. The computer system may alsoinclude an input/output device for providing a physical communicationlink between the computer system and a network, using either a wired ora wireless communication interface.

FIG. 1 is a block diagram of an example of a system 10 upon whichembodiments may be implemented. The system 10 shows the components of anexecution platform for implementing certain functionality of theembodiments. As depicted in FIG. 1, the system 10 includes amicroprocessor 12 (e.g., an Advanced Reduced Instruction Set ComputerMachine, or ARM, processor) coupled to a digital signal processor (DSP)15 via a host interface 11. The host interface 11 translates data andcommands passing between the microprocessor 12 and the DSP 15 into theirrespective formats. In the present embodiment, both the microprocessor12 and the DSP 15 are coupled to a memory 17 via a memory controller 16.In the system 10 embodiment, the memory 17 is a shared memory, wherebythe memory 17 stores instructions and data for both the microprocessor12 and the DSP 15. Access to the shared memory 17 is through the memorycontroller 16. The shared memory 16 also includes a video frame bufferfor storing pixel data that drives a coupled display 18.

As described above, certain processes and steps of the embodiments arerealized, in at least one embodiment, as a series of instructions (e.g.,a software program or programs) that reside within computer-readablememory (e.g., memory 17) of a computer system (e.g., system 10) and areexecuted by the microprocessor 12 and DSP 15 of system 10. Whenexecuted, the instructions cause the system 10 to implement thefunctionality of embodiments as described below. In another embodiment,certain processes and steps of the present invention are realized inhardware.

The descriptions and examples provided herein are discussed in thecontext of video-based data (also referred to as media data ormultimedia data or content), but the present invention is not solimited. For example, embodiments may also be used with image-baseddata, Web page-based data, graphic-based data and the like, andcombinations thereof.

Embodiments of the present invention can be used with Moving PicturesExperts Group (MPEG) compression (encoding) schemes such as MPEG-1,MPEG-2, MPEG-4, and International Telecommunication Union (ITU) encodingschemes such as H.261, H.263 and H.264; however, the present inventionis not so limited. In general, embodiments can be used with encodingschemes that make use of temporal redundancy or motion compensation—inessence, encoding schemes that use motion vectors to increase the amountof compression (the compression ratio).

FIG. 2 illustrates an example of two frames 21 and 22 of image or videodata. In the example of FIG. 2, the frame 21 (also referred to herein asthe first frame or the reference frame) precedes the frame 22 (alsoreferred to herein as the second frame or current frame) in displayorder. Each of the frames 21 and 22 is organized as a plurality ofmacroblocks, exemplified by the macroblock 23. In one embodiment, amacroblock has dimensions of 16 pixels by 16 pixels; however, thepresent invention is not so limited—macroblocks can have dimensionsother than 16×16 pixels. Although FIG. 2 shows a certain number ofmacroblocks, the present invention is not so limited.

In the example of FIG. 2, a motion vector is associated with eachmacroblock. A motion vector has a dimension that describes its length(magnitude) and a dimension that describes its direction (angle). Amotion vector may have a magnitude of zero. For purposes ofillustration, a motion vector that is properly received at a decoder isrepresented herein as an arrow (e.g., arrow 24) within a macroblock.Macroblocks (e.g., macroblock 25) for which an associated motion vectorwas not properly received are indicated in FIG. 2 by shading. The frame22 also includes a macroblock 28 of interest, indicated by an “X,” wherea motion vector was not properly received. A motion vector may not beproperly received if the data describing the motion vector is late,corrupted or missing.

As indicated in FIG. 2, there may be instances in which motion vectorsfor several macroblocks (that is, a slice of macroblocks consisting ofone or more consecutive macroblocks) are not properly received. As willbe seen, according to the embodiments, a motion vector can be estimatedfor each macroblock in the slice for which a motion vector was notproperly received. However, in general, a motion vector can be estimatedfor any macroblock where there is a desire to do so.

In one embodiment, to estimate a motion vector for a macroblock 28 inthe current frame 22, a macroblock 29 in the reference frame 21 isidentified. The macroblock 29 is in the same position within the frame21 as the macroblock 28 of interest is in the frame 22. Accordingly, themacroblock 28 and the macroblock 29 are said to be co-located. Further,a first plurality (window 26) of macroblocks in the current frame 22that neighbor the macroblock 28 is identified, and a second plurality(window 27) of macroblocks in the reference frame 21 that neighbor themacroblock 29 in the reference frame 21 is also identified. In oneembodiment, the window 27 is in the same position within the frame 21 asthe window 26 is in the frame 22. Accordingly, the window 26 and thewindow 27 are also said to be co-located. In general, the term“co-located” is used to describe a region (e.g., a macroblock or awindow of macroblocks) of one frame and a corresponding region inanother frame that are in the same positions within their respectiveframes. A pair of co-located macroblocks 108 and 109 is also indicated;that is, macroblock 108 is at a position within window 27 that is thesame as the position of macroblock 109 within window 26.

In general, according to embodiments, a motion vector can be estimatedfor any macroblock of interest in the frame 22 by considering theproperly received motion vectors associated with macroblocks in thecurrent frame 22 that neighbor the macroblock of interest, and byconsidering the properly received motion vectors associated withmacroblocks in the reference frame 21 that neighbor a macroblock that isco-located with the macroblock of interest.

In one embodiment, the array of macroblocks in the window 26 surroundsthe macroblock 28 of interest. In one such embodiment, the window 26 andthe window 27 each include a 3×3 array of macroblocks. Windows ofdifferent dimensions, including windows that are not square-shaped, canbe selected. Also, a window does not necessarily have to surround themacroblock of interest, in particular for those instances in which themacroblock of interest is at the edge of a frame.

FIG. 3 illustrates two consecutive image frames (a first frame 32 and asecond frame 34) according to one embodiment. In the example of FIG. 3,the second frame 34 follows the first frame 32 in display order. In anMPEG compression scheme, the first frame 32 can correspond to, forexample, an I-frame or a P-frame, and the second frame 34 can correspondto, for example, a P-frame. In general, the first frame 32 and thesecond frame 34 are “inter-coded” (e.g., inter-coded frames are encodeddependent on other frames).

In the example of FIG. 3, an object 33 is located at a certain positionwithin the first frame 32, and the same object 33 is located at adifferent position within the second frame 34. The MPEG compressionscheme works by encoding the differences between frames. A motion vector35 is used as the simplest way of communicating the change in the imagebetween the frames 32 and 34; that is, the image of the object 33 doesnot have to be sent again just because it moved. In a similar manner, amotion vector can be associated with a macroblock in a frame (e.g., themacroblock 23 of FIG. 2).

FIG. 4 is a data flow diagram 40 showing the flow of data from anencoder to a decoder according to one embodiment. In an encoder, anencoding process 42 compresses (encodes) data 41 using an encodingscheme such as MPEG-1, MPEG-2, MPEG-4, H.261, H.263 or H.264. Thecompressed data 43 is sent from the encoder to a decoder (e.g., thesystem 10 of FIG. 1) via the channel 44, which may be a wired orwireless channel. The received data 45 may include both properlyreceived data and corrupted data. Also, some data may also be lostduring transmission or may arrive late at the decoder. The decodingprocess 46 decompresses (reconstructs) the received data 45 to generatethe reconstructed data 47.

FIG. 5 is a flowchart 50 of one embodiment of a motion vectordomain-based temporal error concealment method. Although specific stepsare disclosed in the flowchart 50 of FIG. 5 (as well as in theflowcharts 70, 90, 100, 110 and 130 of FIGS. 7, 9, 10, 11 and 13,respectively), such steps are exemplary. That is, other embodiments maybe formulated by performing various other steps or variations of thesteps recited in the flowcharts 50, 70, 90, 100, 110 and 130. It isappreciated that the steps in the flowcharts 50, 70, 90, 100, 110 and130 may be performed in an order different than presented, and that thesteps in the flowcharts 50, 70, 90, 100, 110 and 130 are not necessarilyperformed in the sequence illustrated.

FIG. 5 is described with reference also to FIG. 6. FIG. 6 shows a 3×3window 63 of macroblocks selected from a reference frame 61, and a 3×3window 64 of macroblocks selected from a current frame 62. It isunderstood that the reference frame 61 and the current frame 62 eachinclude macroblocks in addition to the macroblocks included in thewindows 63 and 64, respectively.

The window 63 and the window 64 are co-located. In the presentembodiment, the macroblock (MB) 68 of interest—that is, the macroblockfor which a motion vector is to be estimated—lies at the center of thewindow 64, but as mentioned above, that does not have to be the case.

It is understood that the windows 63 and 64 can be other than 3×3windows. For instance, 5×5 windows may be used. Also, if the macroblockof interest is along one edge of the current frame 62, then a windowthat is not square in shape (e.g., a 3×2 or a 2×3 window) may be used.

In one embodiment, the reference frame 61 precedes the current frame 62in display order. In another embodiment, the reference frame 61 may be aframe that comes after the current frame 62 in display order; that is,the reference frame 61 may be a “future frame.” In yet anotherembodiment, both the frame preceding the current frame 62 and the futureframe following the current frame 62 may be considered for the errorconcealment methods described herein.

The use of a future frame may introduce delays into the decodingprocess. However, in applications in which delays can be tolerated,motion vectors from a future frame may be used for error concealment.Also, motion vectors from a future frame may be used in instances inwhich the current frame 62 is the first frame in a sequence of frames(e.g., an I-frame).

In overview, one of the objectives of the method of the flowchart 50 isto intelligently select a set 65 of candidate motion vectors from theproperly received motion vectors that are associated with themacroblocks of the frames 61 and 62. In one embodiment, once the set 65of candidate motion vectors is identified, a vector median filter (VMF)66 is applied to the vectors in the set 65. The output of the VMF 66 isan estimated motion vector (MV) 67 for the macroblock 68 of interest.

In one embodiment, in a block 51 of FIG. 5, the windows 63 and 64 areidentified. Correctly received motion vectors associated with themacroblocks in the window 63, and correctly received motion vectorsassociated with the macroblocks in the window 64, are accessed.

In a block 52, in one embodiment, a determination is made as to whethermotion vectors in the reference frame 61 (specifically, in the window63) are eligible to be included in the set 65 of candidate motionvectors. Embodiments of methods used to make this determination aredescribed in conjunction with FIGS. 7, 8, 9 and 10, below.

In a block 53 of FIG. 5, in one embodiment, if motion vectors in thereference frame can be included in the set 65 of candidate motionvectors, then motion vectors from the window 63 and from the window 64are intelligently selected and included in the set 65. Embodiments ofmethods used to select motion vectors from the windows 63 and 64 aredescribed in conjunction with FIGS. 11, 12, 13 and 14, below.

In a block 54 of FIG. 5, in one embodiment, if motion vectors in thereference frame are not eligible to be included in the set 65 ofcandidate motion vectors, then only motion vectors from window 64 areselected and included in the set 65. Note that it is possible that theremay be instances in which the window 64 contains no properly receivedmotion vectors. The method of FIGS. 7 and 8 can be used to address thoseinstances.

In a block 55 of FIG. 5, in one embodiment, a statistical measure of theset 65 of candidate motion vectors is determined. The statisticalmeasure defines a motion vector 67 for the macroblock 68 of interest.The motion vector 67 can then be applied to the macroblock 68 ofinterest.

In one embodiment, the statistical measure is the median of the set 65of candidate motion vectors. In one such embodiment, the median(specifically, the median vector) of the set 65 is determined, asfollows.

For an array of N m-dimensional vectors, V=({overscore (v)}₁,{overscore(v)}₂, . . . ,{overscore (v)}_(N)), with {overscore (v)}₁ε

^(m), for i=1,2, . . . ,N, the median vector {overscore (v)}_(VM) is thevector that satisfies the following constraint:${{{\sum\limits_{i = 1}^{N}{{{\overset{\rightarrow}{v}}_{VM} - {\overset{\rightarrow}{v}}_{i}}}_{p}} \leq {\sum\limits_{i = 1}^{N}{{{\overset{\rightarrow}{v}}_{j} - {\overset{\rightarrow}{v}}_{i}}}_{p}}};\quad{j = 1}},2,\ldots\quad,{N;}$

where p denotes the p-norm metrics between the vectors. For simplicity,in one embodiment, p=1 is used. For a two-dimensional vector {overscore(v)}=(v(x), v(y)), the 1-norm distance between {overscore (v)}₀ and{overscore (v)}₁ is:∥{overscore (v)} ₀ −{overscore (v)} ₁∥_(p=1) =|v ₀(x)−v ₁(x)|+|v ₀(y)−v₁(y)|.

Thus, in one embodiment, the estimated motion vector 67 for themacroblock 68 of interest is the median of the set 65 of candidatemotion vectors. Statistical measures of the set 65 of candidate motionvectors other than the median can be determined and used for theestimated motion vector 67. For example, the average of the set 65 canbe determined and used.

In general, a set 65 of candidate motion vectors is identified. The set65 is then operated on in some manner to determine an estimated motionvector 67 for the macroblock 68 of interest. The estimated motion vector67 may be one of the motion vectors in the set 65, or the estimatedmotion vector 67 may be a motion vector determined by operating on theset 65.

Significantly, the estimated motion vector 67 is determined in themotion vector domain and not in the pixel domain. Specifically, pixelvalues are not used for error concealment, and distortion valuesassociated with each of the candidate motion vectors are not calculatedfor error concealment. Accordingly, computational complexity andassociated decoding delays are reduced. Also, there is no need to accessthe frame buffer to retrieve pixel values, eliminating that source ofadditional decoding delays. Furthermore, by intelligently selectingmotion vectors to be included in the set 65 of candidate motion vectors,peak signal-to-noise ratios (PSNRs) comparable to if not better than thePSNRs associated with pixel-based error concealment techniques areachieved.

FIG. 7 is a flowchart 70 of one embodiment of a method for selectingcandidate motion vectors used in a motion vector domain-based temporalerror concealment process. Flowchart 70 describes one embodiment of amethod for implementing blocks 52, 53 and 54 of FIG. 5. FIG. 7 isdescribed with reference also to FIG. 8.

In one embodiment, in a block 71 of FIG. 7, the window 83 (in areference frame 81) and the window 84 (in the current frame 82) areidentified. It is understood that the reference frame 81 and the currentframe 82 each include macroblocks in addition to the macroblocksincluded in the windows 83 and 84, respectively.

Properly received motion vectors associated with the macroblocks in thewindow 83, and properly received motion vectors associated with themacroblocks in the window 84, can then be accessed. Properly receivedmotion vectors in the window 83 are identified using a letter A, whileproperly received motion vectors in the window 84 are identified using aletter B.

In a block 72, for each pair of co-located macroblocks within thewindows 83 and 84, a determination is made as to whether there is aproperly received motion vector for the macroblock in the window 84.

In a block 73, if there is a properly received motion vector for amacroblock in the window 84, that motion vector is included in the set85 of candidate motion vectors, and the motion vector for the co-locatedmacroblock in the window 83 is not included in the set 85. For example,there is a properly received motion vector for the macroblock 87 (in thewindow 83 in the reference frame 81) and a properly received motionvector for the macroblock 89 (in the window 84 in the current frame 82).According to one embodiment, the motion vector associated with themacroblock 89 (current frame 82) is included in the set 85, and themotion vector associated with the macroblock 87 (reference frame 81) isnot included in the set 85.

In a block 74, if there is not a properly received motion vector for amacroblock in the window 84, then the motion vector for the co-locatedmacroblock in the window 83 is included in the set 85 of candidatemotion vectors. For example, there is not a properly received motionvector for the macroblock 88 of interest, and so the motion vectorassociated with the co-located macroblock 86 (in the reference frame 81)is included in the set 85.

As described above, a statistical measure of the set 85 of candidatemotion vectors is determined (refer to the discussion of FIGS. 5 and 6).

In some instances, motion from one frame to the next frame may not becontinuous. For example, a reference frame may include one type ofmotion, while motion in the current frame may have changed direction orstopped. Furthermore, an object in a reference frame may move out of theneighborhood of a macroblock of interest, and so it may not be suitableto include a motion vector for that object in the set of candidatemotion vectors.

FIG. 9 is a flowchart 90 of one embodiment of a method for detectingframe-to-frame motion change. FIG. 10 is a flowchart 100 of anotherembodiment of a method for detecting frame-to-frame motion change.Either or both of the methods of the flowcharts 90 and 100 can be usedto determine whether motion vectors from a reference frame should beincluded in the set of candidate motion vectors, in order to address thepoints mentioned in the preceding paragraph.

With reference first to FIG. 9, the flowchart 90 describes oneembodiment of a method for implementing the block 52 of FIG. 5. In ablock 91, a first range of values for motion vectors associated with areference frame is determined. In a block 92, a second range of valuesfor motion vectors associated with the current frame is determined. In ablock 93, the first and second ranges of values are compared, and themotion vectors associated with the reference frame are included in theset of candidate motion vectors according to the results of thecomparison.

FIG. 9 is described further with reference also to FIG. 2. In the block91, in one embodiment, motion vector statistics are calculated for theproperly received motion vectors associated with the reference frame 21.

In the block 92, in one embodiment, motion vector statistics arecalculated for the properly received motion vectors associated with thecurrent frame 22.

In one embodiment, all of the motion vectors associated with thereference frame 21 and the current frame 22 are included in thecalculations of motion vector statistics. In another embodiment, onlysubsets of the motion vectors are used instead of all of the motionvectors. In the latter embodiment, for example, the subsets may includeonly the motion vectors associated with macroblocks for which motionvectors for both frames were properly received. That is, for example, amotion vector for a macroblock in the reference frame 21 is onlyincluded in a first subset if the motion vector for the co-locatedmacroblock in the current frame 22 was also properly received.Similarly, a motion vector for a macroblock in the current frame 22 isonly included in a second subset if the motion vector for the co-locatedmacroblock in the reference frame 21 was also properly received.

In one embodiment, for each frame, the statistics calculated include themean and standard deviation of the motion vector dimensions(magnitude/length and direction/angle). Let/be the set of indices of themotion vectors {overscore (v)}; that are included in the calculations ofmotion vector statistics, and let M be the size of the set l. Then themeans and standard deviations (std) for the magnitudes (mag) and angles(ang) are calculated as follows for the reference frame 21 and thecurrent frame 22:${{mean}_{mag\_ frm} = {\frac{1}{M}{\sum\limits_{i \in I}{{mag}\left( {{\overset{\rightarrow}{v}}_{frm}(i)} \right)}}}};$${{mean}_{ang\_ frm} = {\frac{1}{M}{\sum\limits_{i \in I}{{ang}\left( {{\overset{\rightarrow}{v}}_{frm}(i)} \right)}}}};$${{std}_{mag\_ frm} = \sqrt{\frac{1}{m}{\sum\limits_{i \in I}\left( {{mag}\left( {{{\overset{\rightarrow}{v}}_{frm}(i)} - {mean}_{mag\_ frm}} \right)}^{2} \right.}}};{and}$${{std}_{ang\_ frm} = \sqrt{\frac{1}{M}{\sum\limits_{i \in I}\left( {{ang}\left( {{{\overset{\rightarrow}{v}}_{frm}(i)} - {mean}_{ang\_ frm}} \right)}^{2} \right.}}};$

where the subscript “form” refers to either the current frame or thereference frame. Once the means and standard deviations are calculated,the ranges (mean_(mag) _(—) _(frm)−std_(mag) _(—) _(frm), mean_(mag)_(—) _(frm)+std_(mag) _(—) _(frm)) and (mean_(ang) _(—) _(frm)−std_(ang)_(—) _(frm), mean_(ang) _(—) _(frm)+std_(ang) _(—) _(frm)) are formedfor each of the current and reference frames.

In the block 93, in one embodiment, the ranges of the motion vectormagnitudes for the reference frame 21 and for the current frame 22 arecompared, and the ranges of the motion vector angles for the referenceframe 21 and for the current frame 22 are also compared. In oneembodiment, if the range of motion vector magnitudes for the referenceframe 21 overlaps the range of motion vector magnitudes for the currentframe 22, and if the range of motion vector angles for the referenceframe 21 overlaps the range of motion vector angles for the currentframe 22, then the reference frame 21 and the current frame 22 arejudged to have similar motion. Accordingly, motion vectors from thereference frame 21 are eligible for inclusion in the set of candidatemotion vectors (e.g., the set 65 of FIG. 6).

With reference now to FIG. 10, the flowchart 100 describes anotherembodiment of a method for implementing the block 52 of FIG. 5. In ablock 101, the dimensions of pairs of motion vectors are compared todetermine whether motion vectors in each of the pairs are similar toeach other. Each of the pairs of motion vectors includes a first motionvector associated with a first macroblock at a position in a referenceframe, and a second motion vector associated with a second macroblock atthe position in the current frame.

In a block 102, the number of pairs of motion vectors in the referenceand current frames that are similar is counted. In a block 103, motionvectors from the reference frame are eligible for inclusion in the setof candidate motion vectors if the number exceeds a threshold.

FIG. 10 is described further with reference also to FIG. 2. In the block101, in one embodiment, the dimensions of each pair of co-locatedmacroblocks are compared. The macroblocks 108 and 109 of FIG. 2 are anexample of a pair of co-located macroblocks.

In one embodiment, to facilitate the comparison, each received motionvector in the reference frame 21 and each received motion vector in thecurrent frame 22 is given a magnitude label and a direction label. Inone such embodiment, the magnitude label has a value of either zero (0)or one (1), depending on its relative magnitude. For example, a motionvector having a magnitude of less than or equal to two (2) pixels isassigned a magnitude label of 0, and a motion vector having a magnitudeof more than 2 pixels is assigned a magnitude label of 1. In oneembodiment, the direction label has a value of 0, 1, 2 or three (3). Forexample, relative to a vertical line in a frame, a motion vector havingan angle greater than or equal to −45 degrees but less than 45 degreescould be assigned a direction label of 0, a motion vector having anangle greater than or equal to 45 degrees but less than 135 degreescould be assigned a direction label of 1, and so on. Other schemes forlabeling the magnitude and direction of motion vectors can be used.

In one embodiment, for each pair of co-located macroblocks, themagnitude labels of the 2 motion vectors in the pair are compared, andthe direction labels of the 2 motion vectors in the pair are compared.In one embodiment, if the magnitude labels are the same and thedirection labels are not opposite for the 2 motion vectors in a pair,then that pair of motion vectors is defined as being similar. Note that,in the present embodiment, the direction labels do not necessarily haveto be the same in order for the 2 motion vectors in a pair to beconsidered similar. For example, using the scheme described above, adirection label of 0 would be considered similar to a direction label of0, 1 or 3, but opposite to a direction label of 2. Other rules definingwhat constitutes similar motion vectors can be used.

In the block 102 of FIG. 10, in one embodiment, the number of pairs ofco-located macroblocks that contain similar motion vectors is counted.In other words, the number of pairs of similar motion vectors iscounted.

In the block 103, in one embodiment, motion vectors from the referenceframe 21 are eligible for inclusion in the set of candidate motionvectors (e.g., the set 65 of FIG. 6) if the count made in the block 102exceeds a threshold. In one embodiment, the threshold is equal toone-half of the number of macroblocks in either of the two frames 21 or22.

In the neighborhood of a macroblock of interest, there may be a motionboundary—objects on one side of a motion boundary may move differentlyfrom objects on the other side of the motion boundary. FIG. 11 is aflowchart 110 of one embodiment of a method for locating a motionboundary. The flowchart 110 describes one embodiment of a method ofimplementing the block 53 of FIG. 5. Note that, in one embodiment, theblock 53 (and hence the method of the flowchart 110) is implementeddepending on the outcome of the block 52 of FIG. 5.

In a block 111 of FIG. 11, a motion boundary is identified in areference frame. In a block 112, the set of candidate motion vectorsincludes only those motion vectors that are associated with macroblocksin the reference frame that lie on the same side of the motion boundaryas a macroblock in the reference frame that is co-located with amacroblock of interest in the current frame.

FIG. 11 is described further with reference also to FIG. 12. FIG. 12shows a window 125 in a reference frame 121, and a window 126 in acurrent frame 122. It is understood that the reference frame 121 and thecurrent frame 122 each include macroblocks in addition to themacroblocks included in the windows 125 and 126, respectively.

In one embodiment, in the block 111, a motion boundary 129 is identifiedin the reference frame 121. In one embodiment, the motion boundary 129is identified in the following manner. Each of the motion vectorsassociated with the macroblocks in the window 125 in the reference frame121 is assigned a magnitude label and a direction label. The discussionabove in conjunction with FIG. 10 describes one method for labelingmotion vectors.

The motion vector associated with the macroblock 124 in the referenceframe 121 that is at the same position as the macroblock 123 of interestin the current frame 122 is classified as class 0. That is, themacroblock 124 is co-located with the macroblock 123 of interest, and assuch, the motion vector associated with the macroblock 124 is identifiedas being the first member of a particular class (e.g., class 0).

The magnitude labels of the other motion vectors associated with thewindow 125 are each compared to the magnitude label of the motion vectorassociated with the macroblock 124, and the direction labels of theother motion vectors in the window 125 are each compared to thedirection label of the motion vector associated with the macroblock 124.

In one embodiment, if the magnitude label for a motion vector is thesame as that of the motion vector associated with the macroblock 124,and if the angle label for that motion vector is not opposite that ofthe motion vector associated with the macroblock 124, then that motionvector is defined as being similar to the motion vector associated withthe macroblock 124, and that motion vector is also classified as class0. As mentioned, the process just described is repeated for each motionvector associated with the window 125, to generate the local motionclass map 127.

In one embodiment, in the block 112, only those motion vectorsassociated with the window 125 that are in the same class as the motionvector associated with the macroblock 124 are included in the set 126 ofcandidate motion vectors. In other words, in the present embodiment,only the motion vectors in the window 125 in the reference frame 121that are on the same side of the motion boundary 129 as the macroblock124 (the macroblock co-located with the macroblock 123 of interest) areincluded in the set 128 of candidate motion vectors. That is, in theexample of FIG. 12, only the motion vectors classified as class 0 areincluded in the set 128. As described above, a statistical measure ofthe set 128 of candidate motion vectors is then determined (refer to thediscussion of FIGS. 5 and 6).

Note that properly received motion vectors associated with the window126 of the current frame 122 can also be included in the set 128 if theyare associated with macroblocks that also lie on the same side of themotion boundary as the macroblock 123 of interest. For example, afterthe map 127 is determined, the macroblocks in the window 126 that areco-located with those macroblocks in the window 127 that are classifiedas class 0 can also be classified as class 0, and the motion vectorsassociated with those macroblocks in the window 126 can be included inthe set 128.

FIG. 13 is a flowchart 130 of one embodiment of a method that uses thetrajectory of a moving object to select a candidate motion vector. Theflowchart 130 describes one embodiment of a method of implementing theblock 53 of FIG. 5.

In a block 131 of FIG. 13, an object in a first macroblock in areference frame is identified. In a block 132, a motion vectorassociated with the object is included in the set of candidate motionvectors if the object sufficiently overlaps a co-located secondmacroblock in the current frame (that is, the first macroblock and thesecond macroblock are in the same position within their respectiveframes).

FIG. 13 is described further with reference also to FIG. 14. FIG. 14shows a window 147 of a reference frame 141 and a window 148 of acurrent frame 142. The macroblock 143 is co-located with the macroblock146. It is understood that the reference frame 141 and the current frame142 each include macroblocks in addition to the macroblocks included inthe windows 147 and 148, respectively.

In the block 131, in one embodiment, an object 144 within the referenceframe, and associated with the macroblock 143 that is co-located withthe macroblock 146, is identified. In the current frame 142, the object144 has moved to a different position, and is now associated with amacroblock 145.

In the block 132, in one embodiment, a determination is made as towhether the macroblock 145 that contains the object 144 overlaps themacroblock 146 by a sufficient amount. If so, the motion vectorassociated with the object 144 can be included in the set of candidatemotion vectors (e.g., the set 65 of FIG. 6). If not, the motion vectorassociated with the object 144 is not included in the set.

Note that method described in conjunction with FIGS. 13 and 14 can besimilarly applied to any of the macroblocks within the windows 147 and148. That is, although described for the center macroblock of thewindows 147 and 148, the present invention is not so limited.

In one embodiment, an overlap of greater than or equal to 25 percent isconsidered sufficient. Various techniques can be used to determinewhether the macroblock 145 overlaps the macroblock 146 by that amount.In one embodiment, the macroblocks 145 and 146 are each associated witha set of two-dimensional coordinates that define their respectivepositions within the current frame 142. Using these coordinates, forexample, the corners of one of the macroblocks 145 and 146 can becompared to the midpoints of the sides of the other macroblock todetermine whether the amount of overlap exceeds 25 percent. Thresholdsother than 25 percent can be used.

The embodiments of FIGS. 9-14 have been described separately in order tomore clearly describe certain aspects of the embodiments; however, it isappreciated that the embodiments may be implemented by combiningdifferent aspects of these embodiments. In one embodiment, one of themethods described in conjunction with FIGS. 9 and 10 is combined withone of the methods described in conjunction with FIGS. 11-14.

In summary, embodiments in accordance with the present invention providemethods and systems for temporal error concealment using motion vectorsin the motion vector domain rather than pixel values in the pixeldomain. Accordingly, computational complexity is reduced becausedistortion evaluations can be eliminated with regard to errorconcealment; the number of computation steps may be reduced by as muchas 85 percent. Decoding delays are reduced from one frame to one sliceof macroblocks; that is, in order to use neighboring motion vectors toestimate an absent motion vector, processing of only a slice (e.g., onerow) of macroblocks may be delayed. Memory access times, and associateddecoding delays, are reduced because memory accesses to retrieve pixelvalues can be eliminated with regard to error concealment. Yet theembodiments described herein yield PSNRs that are comparable to if notbetter than PSNRs associated with pixel-based error concealmenttechniques. Furthermore, embodiments can be implemented without havingto make hardware changes.

The concepts described herein can be used for applications other thanerror concealment. For instance, embodiments can be used in motionestimation at an encoder. For example, in conventional hierarchicalmotion estimation, motion vectors found in the lowest spatial resolutionare used as initial estimates of motion vectors for higher resolutions.Instead, motion vectors selected as described above can be used as theinitial estimates to speed up motion estimation at the encoder.

Embodiments of the present invention are thus described. While thepresent invention has been described by the various differentembodiments, it should be appreciated that the present invention shouldnot be construed as limited by such embodiments, but rather construedaccording to the below claims.

1. A method of processing video data, said method comprising: selectinga set of motion vectors from a first plurality of motion vectorsassociated with a first plurality of macroblocks in a first frame ofsaid video data and from a second plurality of motion vectors associatedwith a second plurality of macroblocks in a second frame of said videodata; determining a statistical measure of said set of motion vectors,said statistical measure defining a motion vector for a macroblock ofinterest in said second plurality of macroblocks; and applying saidmotion vector to said macroblock of interest.
 2. The method of claim 1wherein said determining comprises determining the median of said set,wherein said motion vector for said macroblock of interest is saidmedian.
 3. The method of claim 1 wherein said determining comprisesdetermining the average of said set, wherein said motion vector for saidmacroblock of interest is said average.
 4. The method of claim 1 whereinsaid set comprises: said second plurality of motion vectors; and motionvectors for selected macroblocks of said first plurality of macroblocks,wherein if there is a first motion vector for a first macroblock at aposition in said first frame and also a second motion vector for asecond macroblock at said position in said second frame, said secondmotion vector is included in said set and said first motion vector isnot included in said set.
 5. The method of claim 1 further comprisingdeciding whether motion vectors from said first plurality of motionvectors are eligible for inclusion in said set.
 6. The method of claim 5wherein said deciding comprises: determining a first range of values formotion vectors associated with said first frame; determining a secondrange of values for motion vectors associated with said second frame;and comparing said first and second ranges of values, wherein motionvectors from said first plurality of motion vectors are eligible forinclusion in said set if said first and second ranges overlap by aspecified amount.
 7. The method of claim 5 wherein said decidingcomprises: comparing dimensions of pairs of motion vectors to determinewhether motion vectors in each of said pairs are similar to each other,each of said pairs of motion vectors comprising a first motion vectorassociated with a first macroblock at a position in said first frame anda second motion vector associated with a second macroblock at saidposition in said second frame, wherein motion vectors are similarprovided they satisfy a rule; and counting the number of pairs of motionvectors in said first and second frames that are similar, wherein motionvectors from said first plurality of motion vectors are eligible forinclusion in said set if said number exceeds a threshold.
 8. The methodof claim 5 wherein said set comprises motion vectors associated withmacroblocks of said first plurality of macroblocks that lie on the sameside of a motion boundary as a macroblock in said first frame at thesame position as said macroblock of interest.
 9. The method of claim 5wherein said set comprises motion vectors from said first plurality ofmotion vectors that are similar to a motion vector for a macroblock insaid first plurality of macroblocks that is at the same position as saidmacroblock of interest, wherein motion vectors are similar provided theysatisfy a rule.
 10. The method of claim 1 wherein a motion vectorassociated with an object that is contained in a first macroblock insaid first frame at the same position as a second macroblock in saidsecond frame is included in said set provided that in said second framesaid object overlaps said second macroblock by a specified amount. 11.The method of claim 1 wherein said first frame precedes said secondframe in order of display.
 12. A computer-usable medium havingcomputer-readable program code embodied therein for causing a decodingdevice to perform a video data processing method comprising: selecting aset of motion vectors from a first plurality of motion vectorsassociated with a first plurality of macroblocks in a first frame ofvideo data and from a second plurality of motion vectors associated witha second plurality of macroblocks in a second frame of said video data;determining a statistical measure of said set of motion vectors, saidstatistical measure defining a motion vector for a macroblock ofinterest in said second plurality of macroblocks; and applying saidmotion vector to said macroblock of interest.
 13. The computer-usablemedium of claim 12 wherein said set comprises: said second plurality ofmotion vectors; and motion vectors for selected macroblocks of saidfirst plurality of macroblocks, wherein if there is a first motionvector for a first macroblock at a position in said first frame and alsoa second motion vector for a second macroblock at said position in saidsecond frame, said second motion vector is included in said set and saidfirst motion vector is not included in said set.
 14. The computer-usablemedium of claim 12 wherein said computer-readable program code embodiedtherein causes said decoding device to perform said video dataprocessing method further comprising: determining a first range ofvalues for motion vectors associated with said first frame; determininga second range of values for motion vectors associated with said secondframe; and comparing said first and second ranges of values, whereinmotion vectors from said first plurality of motion vectors are eligiblefor inclusion in said set if said first and second ranges overlap by aspecified amount.
 15. The computer-usable medium of claim 12 whereinsaid computer-readable program code embodied therein causes saiddecoding device to perform said video data processing method furthercomprising: comparing dimensions of pairs of motion vectors to determinewhether motion vectors in each of said pairs are similar to each other,each of said pairs of motion vectors comprising a first motion vectorassociated with a first macroblock at a position in said first frame anda second motion vector associated with a second macroblock at saidposition in said second frame, wherein motion vectors are similarprovided they satisfy a rule; and counting the number of pairs of motionvectors in said first and second frames that are similar, wherein motionvectors from said first plurality of motion vectors are eligible forinclusion in said set if said number exceeds a threshold.
 16. Thecomputer-usable medium of claim 12 wherein said set comprises motionvectors associated with macroblocks of said first plurality ofmacroblocks that lie on the same side of a motion boundary as amacroblock in said first frame at the same position as said macroblockof interest.
 17. The computer-usable medium of claim 12 wherein a motionvector associated with an object that is contained in a first macroblockin said first frame at the same position as a second macroblock in saidsecond frame is included in said set provided that in said second framesaid object overlaps said second macroblock by a specified amount. 18.The computer-usable medium of claim 12 wherein said first frame precedessaid second frame in order of display.
 19. A system for processing videodata, said system comprising: means for selecting a set of motionvectors from a first plurality of motion vectors associated with a firstplurality of macroblocks in a first frame of said video data and from asecond plurality of motion vectors associated with a second plurality ofmacroblocks in a second frame of said video data; means for determininga statistical measure of said set of motion vectors, said statisticalmeasure defining a motion vector for a macroblock of interest in saidsecond plurality of macroblocks; and applying said motion vector to saidmacroblock of interest.
 20. The system of claim 19 wherein said setcomprises: said second plurality of motion vectors; and motion vectorsfor selected macroblocks of said first plurality of macroblocks, whereinif there is a first motion vector for a first macroblock at a positionin said first frame and also a second motion vector for a secondmacroblock at said position in said second frame, said second motionvector is included in said set and said first motion vector is notincluded in said set.
 21. The system of claim 19 further comprisingmeans for deciding whether motion vectors from said first plurality ofmotion vectors are eligible for inclusion in said set.
 22. The system ofclaim 21 wherein said means for deciding comprises: means fordetermining a first range of values for motion vectors associated withsaid first frame; means for determining a second range of values formotion vectors associated with said second frame; and means forcomparing said first and second ranges of values, wherein motion vectorsfrom said first plurality of motion vectors are eligible for inclusionin said set if said first and second ranges overlap by a specifiedamount.
 23. The system of claim 21 wherein said means for decidingcomprises: means for comparing dimensions of pairs of motion vectors todetermine whether motion vectors in each of said pairs are similar toeach other, each of said pairs of motion vectors comprising a firstmotion vector associated with a first macroblock at a position in saidfirst frame and a second motion vector associated with a secondmacroblock at said position in said second frame, wherein motion vectorsare similar provided they satisfy a rule; and means for counting thenumber of pairs of motion vectors in said first and second frames thatare similar, wherein motion vectors from said first plurality of motionvectors are eligible for inclusion in said set if said number exceeds athreshold.
 24. The system of claim 21 wherein said set comprises motionvectors associated with macroblocks of said first plurality ofmacroblocks that lie on the same side of a motion boundary as amacroblock in said first frame at the same position as said macroblockof interest.
 25. The system of claim 21 wherein said set comprisesmotion vectors from said first plurality of motion vectors that aresimilar to a motion vector for a macroblock in said first plurality ofmacroblocks that is at the same position as said macroblock of interest,wherein motion vectors are similar provided they satisfy a rule.
 26. Thesystem of claim 19 wherein a motion vector associated with an objectthat is contained in a first macroblock in said first frame at the sameposition as a second macroblock in said second frame is included in saidset provided that in said second frame said object overlaps said secondmacroblock by a specified amount.
 27. The system of claim 19 whereinsaid first frame precedes said second frame in order of display.
 28. Adevice comprising: a microprocessor; and a memory unit coupled to saidmicroprocessor, said memory unit containing instructions that whenexecuted by said microprocessor implement a method for processing videodata, said method comprising: selecting a set of motion vectors from afirst plurality of motion vectors associated with a first plurality ofmacroblocks in a first frame of said video data and from a secondplurality of motion vectors associated with a second plurality ofmacroblocks in a second frame of said video data; determining astatistical measure of said set of motion vectors, said statisticalmeasure defining a motion vector for a macroblock of interest in saidsecond plurality of macroblocks; and applying said motion vector to saidmacroblock of interest.
 29. The device of claim 28 wherein said setcomprises: said second plurality of motion vectors; and motion vectorsfor selected macroblocks of said first plurality of macroblocks, whereinif there is a first motion vector for a first macroblock at a positionin said first frame and also a second motion vector for a secondmacroblock at said position in said second frame, said second motionvector is included in said set and said first motion vector is notincluded in said set.
 30. The device of claim 28 wherein said methodfurther comprises: determining a first range of values for motionvectors associated with said first frame; determining a second range ofvalues for motion vectors associated with said second frame; andcomparing said first and second ranges of values, wherein motion vectorsfrom said first plurality of motion vectors are eligible for inclusionin said set if said first and second ranges overlap by a specifiedamount.
 31. The device of claim 28 wherein said method furthercomprises: comparing dimensions of pairs of motion vectors to determinewhether motion vectors in each of said pairs are similar to each other,each of said pairs of motion vectors comprising a first motion vectorassociated with a first macroblock at a position in said first frame anda second motion vector associated with a second macroblock at saidposition in said second frame, wherein motion vectors are similarprovided they satisfy a rule; and counting the number of pairs of motionvectors in said first and second frames that are similar, wherein motionvectors from said first plurality of motion vectors are eligible forinclusion in said set if said number exceeds a threshold.
 32. The deviceof claim 28 wherein said set comprises motion vectors associated withmacroblocks of said first plurality of macroblocks that lie on the sameside of a motion boundary as a macroblock in said first frame at thesame position as said macroblock of interest.
 33. The device of claim 28wherein said set comprises motion vectors from said first plurality ofmotion vectors that are similar to a motion vector for a macroblock insaid first plurality of macroblocks that is at the same position as saidmacroblock of interest, wherein motion vectors are similar provided theysatisfy a rule.
 34. The device of claim 28 wherein a motion vectorassociated with an object that is contained in a first macroblock insaid first frame at the same position as a second macroblock in saidsecond frame is included in said set provided that in said second framesaid object overlaps said second macroblock by a specified amount. 35.The device of claim 28 wherein said first frame precedes said secondframe in order of display.